Image processing apparatus, image capturing apparatus, image processing method, and storage medium

ABSTRACT

There is provided an image processing apparatus. An obtaining unit obtains a first image to which tone conversion processing conforming to a first input/output characteristic having a first maximum output luminance value has been applied. A generation unit generates first correction information for correcting a luminance value of the first image based on a difference regarding an output luminance value between the first input/output characteristic and a second input/output characteristic having a second maximum output luminance value, and on second correction information for correcting a luminance value of a second image to which tone conversion processing conforming to the second input/output characteristic has been applied. A correction unit corrects a luminance value of the first image in conformity to the first correction information.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to an image processing apparatus, an imagecapturing apparatus, an image processing method, and a storage medium.

Description of the Related Art

In recent years, for example, the evolution of LED elements in displayshas made it possible to display high-dynamic-range (hereinafter referredto as “HDR”) image data as is without compressing the same. According toHDR, image presentation that exploits a high dynamic range is possible,and thus the colors and details of a high-luminance range that cannot bepresented according to a conventional dynamic range (hereinafterreferred to as “SDR”) can be reproduced more authentically.

Along with this widespread use of HDR, there is demand for imagecreation that is suited for a subject and a scene in the case of HDR,similarly to SDR. For example, image creation that achieves brightnessto reproduce a transparent skin color is preferred in a portrait scene,whereas image creation that makes blue sky and green vivid is preferredin landscape shooting. To realize these image presentations, the needarises to apply some sort of color/luminance correction to an originalimage signal.

There are two types of HDR: a PQ (Perceptual Quantization) methodstandardized in SMPTE ST2084, and an HLG (Hybrid Log Gamma) methoddeveloped by ARM STD-B67. A major difference between these two methodsis that, while the HLG method treats luminance values in a relativemanner similarly to SDR, the PQ method treats them as absoluteluminances with a maximum of 10000 nits. Due to this difference, whenshooting has been performed using the PQ method in a shooting mode inwhich an output dynamic range (D range) changes, a peak luminance at thetime of presentation on a display changes. Hereinafter, it will beassumed that a description of HDR is based on the premise that the PQmethod is used.

In FIG. 4, gamma curves 41, 42 represent examples of input/outputcharacteristics that correspond to two types of shooting modes withdifferent output D ranges. A horizontal axis represents the number ofinput stages, and a vertical axis represents an output luminance.Comparing the gamma curves of the respective shooting modes with eachother, although they have the same input/output characteristic in aluminance range until the curves start to lie flat, they have differentinput/output characteristics in a range of luminances higher than thatluminance range. As a result, the gamma curves 41, 42 respectively havedifferent peak luminances 43, 44.

Returning to the description of color/luminance correction of HDR, FIGS.5A to 5D show conceptual diagrams for a case where certaincolor/luminance correction has been applied to each of the images thathave been developed in the two types of shooting modes shown in FIG. 4.FIG. 5A is a conceptual diagram of correction effects on a certain inputluminance range in a shooting mode with a low peak luminance inconnection with the certain color/luminance correction, and FIG. 5B is aconceptual diagram of correction effects on a lower input luminancerange in connection with the same color/luminance correction. FIG. 5C isa conceptual diagram of correction effects on the same input luminancerange as FIG. 5A in a shooting mode with a high peak luminance inconnection with the same color/luminance correction, and FIG. 5D is aconceptual diagram of ideal correction effects in the shooting mode ofFIG. 5C. In FIG. 5C, although the same correction has been applied tothe same input luminance range as in FIG. 5A, a deficiency in correctionamounts is sensed due to the difference in the output D range.Therefore, in order to achieve correction effects that are equivalent tocorrection effects achieved in a shooting mode with a low peak luminancealso in a shooting mode with a high peak luminance, it is necessary toachieve the effects of FIG. 5D by applying correction that is equivalentto correction to an input luminance range lower than a target inputluminance range shown in FIG. 5B.

As described above, according to HDR, as there is a case where adifference in a peak luminance arises depending on a shooting mode,there is a case where a deficiency or an excess in correction amountsshown in FIGS. 5A to 5D occurs if the same correction is applied indisregard of such a difference. This is because, when a peak luminanceis high, tone properties are enhanced and thus color reproduction in ahigh-luminance range is improved compared to when a peak luminance islow. Therefore, even with the same input signal, the brightness andchroma differ depending on a shooting mode, thereby generating adifference in correction effects as well. Consequently, in order toachieve appropriate correction effects in any shooting mode, the needarises to change correction amounts in accordance with a change in apeak luminance.

Referring to Japanese Patent No. 4878008, it discloses abrightness/chroma/hue correction method that enables appropriate colorreproduction even when the presentable gamut differs depending on anoutput device. Next, referring to Japanese Patent Laid-Open No.2018-026606, it discloses a color/luminance correction approach forreproducing original tone properties of HDR when an image obtainedthrough HDR shooting is displayed on an SDR monitor.

Japanese Patent No. 4878008 does not describe HDR image output. On theother hand, the correction approach of Japanese Patent Laid-Open No.2018-026606 is the correction approach at the time of compressing a peakluminance from an HDR luminance value to an SDR luminance value, and animage that is output using this approach is not an HDR image but an SDRimage. Conventionally, a technique to effectively apply luminancecorrection for an HDR image in accordance with a change in an output Drange has been unknown.

SUMMARY OF THE INVENTION

The present invention has been made in view of the aforementionedsituations, and provides a technique to enable luminance correction inaccordance with a maximum output luminance value of an input/outputcharacteristic of tone conversion processing that was applied to animage to be corrected.

According to a first aspect of the present invention, there is providedan image processing apparatus comprising at least one processor and/orat least one circuit which function as: an obtaining unit configured toobtain a first image to which tone conversion processing conforming to afirst input/output characteristic having a first maximum outputluminance value has been applied; a generation unit configured togenerate first correction information for correcting a luminance valueof the first image based on a difference regarding an output luminancevalue between the first input/output characteristic and a secondinput/output characteristic having a second maximum output luminancevalue, and on second correction information for correcting a luminancevalue of a second image to which tone conversion processing conformingto the second input/output characteristic has been applied; and acorrection unit configured to correct a luminance value of the firstimage in conformity to the first correction information.

According to a second aspect of the present invention, there is providedan image capturing apparatus, comprising: the image processing apparatusaccording to the first aspect; and at least one processor and/or atleast one circuit which function as: an image capturing unit; and animage generation unit configured to generate the first image by applyingthe tone conversion processing conforming to the first input/outputcharacteristic to an image generated by the image capturing unit.

According to a third aspect of the present invention, there is providedan image processing method executed by an image processing apparatus,comprising: obtaining a first image to which tone conversion processingconforming to a first input/output characteristic having a first maximumoutput luminance value has been applied; generating first correctioninformation for correcting a luminance value of the first image based ona difference regarding an output luminance value between the firstinput/output characteristic and a second input/output characteristichaving a second maximum output luminance value, and on second correctioninformation for correcting a luminance value of a second image to whichtone conversion processing conforming to the second input/outputcharacteristic has been applied; and correcting a luminance value of thefirst image in conformity to the first correction information.

According to a fourth aspect of the present invention, there is provideda non-transitory computer-readable storage medium which stores a programfor causing a computer to execute an image processing method comprising:obtaining a first image to which tone conversion processing conformingto a first input/output characteristic having a first maximum outputluminance value has been applied; generating first correctioninformation for correcting a luminance value of the first image based ona difference regarding an output luminance value between the firstinput/output characteristic and a second input/output characteristichaving a second maximum output luminance value, and on second correctioninformation for correcting a luminance value of a second image to whichtone conversion processing conforming to the second input/outputcharacteristic has been applied; and correcting a luminance value of thefirst image in conformity to the first correction information.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram for describing the development of a RAW imageaccording to a first embodiment.

FIG. 2 is a cross-sectional diagram showing the arrangement of mainlyoptical members, sensors, and the like of a digital camera, which is oneexample of an image processing apparatus.

FIG. 3 is a block diagram showing exemplary configurations of electricalcircuits of a camera main body 1 and an interchangeable lens 2.

FIG. 4 is a diagram showing examples of input/output characteristicswith different output D ranges.

FIGS. 5A to 5D are diagrams for describing a deficiency or an excess incorrection amounts attributed to a difference in a peak luminance.

FIG. 6 is a diagram showing the PQ-EOTF.

FIG. 7 is a diagram for describing processing for generating a compositeLUT.

FIGS. 8A and 8B are diagrams showing a difference in an output luminancevalue between a reference input/output characteristic and a selectedinput/output characteristic.

FIG. 9 is a diagram showing a difference between a reference LUT and thecomposite LUT according to the first embodiment.

FIGS. 10A and 10B are flowcharts of processing for generating adifference LUT according to the first embodiment.

FIGS. 11A and 11B are diagrams for describing the processing forgenerating the difference LUT according to the first embodiment.

FIG. 12 is a diagram for describing the development of a RAW imageaccording to a second embodiment.

FIGS. 13A and 13B are diagrams showing a difference in an inclinationbetween a reference input/output characteristic and a selectedinput/output characteristic.

FIG. 14 is a diagram showing a difference between a reference LUT and acomposite LUT according to the second embodiment.

FIGS. 15A and 15B are flowcharts of processing for generating adifference LUT according to the second embodiment.

FIGS. 16A and 16B are diagrams for describing the processing forgenerating the difference LUT according to the second embodiment.

DESCRIPTION OF THE EMBODIMENTS

Hereinafter, embodiments of the present invention will be described withreference to the attached drawings. Elements that are given the samereference numerals throughout all of the attached drawings represent thesame or similar elements. Note that the technical scope of the presentinvention is defined by the claims, and is not limited by the followingrespective embodiments. Also, not all of the combinations of the aspectsthat are described in the embodiments are necessarily essential to thepresent invention. Also, the aspects that are described in theindividual embodiments can be combined as appropriate.

Note that in the following embodiments, a description will be given of adigital camera, which serves as one example of an image processingapparatus. However, the following embodiments are not limited to adevice that mainly aims to perform shooting, like a digital camera. Forexample, the following embodiments are applicable to any device thatincludes an image processing apparatus built therein or is connected toan external image processing apparatus, like a mobile telephone, apersonal computer (e.g., a laptop type, a desktop type, and a tablettype), and a game device.

First Embodiment

FIG. 2 is a cross-sectional diagram showing the arrangement of mainlyoptical members, sensors, and the like of a digital camera, which is oneexample of an image processing apparatus. The digital camera of thepresent embodiment is a so-called digital single-lens reflex camera ofan interchangeable lens type, and includes a camera main body 1 and aninterchangeable lens 2. In the camera main body 1, an image sensor 10is, for example, a CMOS image sensor or a CCD image sensor, and aplurality of pixels (storage-type photoelectric conversion elements) arearrayed therein. A mechanical shutter 11, which is provided in thevicinity of the front of the image sensor 10, controls an exposuretiming and an exposure period of the image sensor 10. Asemi-transmissive main mirror 3 and a first reflective mirror 7 arrangedon the back side of the main mirror 3 are flipped up at the time ofshooting. A second reflective mirror 8 further reflects a light beamreflected by the first reflective mirror 7, and makes the light beamincident on an AF sensor 9 (a sensor for focus detection). The AF sensor9 may be, for example, an image sensor that has a smaller number ofpixels than the image sensor 10. The first reflective mirror 7, thesecond reflective mirror 8, and the AF sensor 9 are constituents forperforming focus detection using a phase difference detection method atan arbitrary position inside a shooting screen. An AE sensor 6 (a sensorfor metering) receives light of an image on the shooting screenreflected by a pentaprism 4 and a third reflective mirror 5. The AEsensor 6 can divide a light receiving portion into a plurality ofregions, and output luminance information of a subject on aregion-by-region basis. There is no limitation on the division number.Note that in the image sensor, for example, an amplifier circuit for apixel signal and a peripheral circuit for signal processing are formed,in addition to the pixels arranged in the light receiving portion. Thepentaprism 4 constitutes a finder optical system. Although notillustrated in FIG. 2, a subject image reflected by the pentaprism 4 canbe observed from an eyepiece. Among the light rays that have beenreflected by the main mirror 3 and diffused by a focusing screen 12, aportion that is outside an optical axis becomes incident on the AEsensor 6. The interchangeable lens 2 performs information communicationwith the camera main body 1 as necessary via a contact point of a lensmount provided in the camera main body 1. Note that at the time oflive-view display and at the time of moving image recording, the mainmirror 3 is always in a flipped-up state, and thus exposure control andfocus adjustment control are performed using image information of animage capturing surface.

FIG. 3 is a block diagram showing exemplary configurations of electricalcircuits of the camera main body 1 and the interchangeable lens 2 shownin FIG. 2. In the camera main body 1, a camera control unit 21 is asingle-chip microcomputer that includes, for example, an ALU (ARITHMETICand Logic Unit), a ROM, a RAM, an A/D converter, a timer, a serialcommunication port (SPI), and the like built therein. The camera controlunit 21 controls the operations of the camera main body 1 and theinterchangeable lens 2 by, for example, executing programs stored in theROM. Specific operations of the camera control unit 21 will be describedlater.

Output signals from the AF sensor 9 and the AE sensor 6 are connected toan A/D converter input terminal of the camera control unit 21. A signalprocessing circuit 25 controls the image sensor 10 in accordance with aninstruction from the camera control unit 21, applies A/D conversion andsignal processing to a signal output from the image sensor 10, andobtains an image signal. Furthermore, in recording the obtained imagesignal, the signal processing circuit 25 performs necessary imageprocessing, such as compression and composition. A memory 28 is a DRAMor the like, and it is used as a working memory when the signalprocessing circuit 25 performs various types of signal processing, andis used as a VRAM when an image is displayed on a display device 27,which will be described later. The display device 27 is, for example, aback-surface liquid crystal display or an external display that isconnected to the camera main body 1 in conformity to the standards ofHDMI™ or the like. The display device 27 displays information, such assetting values of the digital camera, a message, and a menu screen, andcaptured images. The display device 27 is controlled by an instructionfrom the camera control unit 21. A storage unit 26 is, for example, anon-volatile memory, such as a flash memory, and a captured image signalis input thereto from the signal processing circuit 25.

Under control of the camera control unit 21, a motor 22 flips up anddown the main mirror 3 and the first reflective mirror 7, and chargesthe mechanical shutter 11. An operation unit 23 is an input device groupincluding, for example, switches that are used by a user to operate thedigital camera. The operation unit 23 includes, for example, a releaseswitch for issuing instructions for starting a shooting preparationoperation and starting shooting, a shooting mode selection switch forselecting a shooting mode, directional keys, and an enter key. A contactpoint unit 29 is a contact point for performing communication with theinterchangeable lens 2, and an input/output signal of the serialcommunication port of the camera control unit 21 is connected thereto. Ashutter driving unit 24 is connected to an output terminal of the cameracontrol unit 21, and drives the mechanical shutter 11.

The interchangeable lens 2 includes a contact point unit 50 that ispaired with the contact point unit 29. A lens control unit 51, which isa single-chip microcomputer similar to the camera control unit 21, isconnected to the contact point unit 50, and the lens control unit 51 canperform communication with the camera control unit 21. The lens controlunit 51 controls the operations of the interchangeable lens 2 based onan instruction from the camera control unit 21 by executing programsstored in, for example, the ROM. The lens control unit 51 also notifiesthe camera control unit 21 of information of, for example, the state ofthe interchangeable lens 2. A focusing lens driving unit 52 is connectedto an output terminal of the lens control unit 51, and drives a focusinglens. A zoom driving unit 53 changes the angle of view of theinterchangeable lens under control of the lens control unit 51. Adiaphragm driving unit 54 adjusts an aperture size of a diaphragm undercontrol of the lens control unit 51.

When the interchangeable lens 2 is attached to the camera main body 1,the lens control unit 51 and the camera control unit 21 can perform datacommunication with each other via the contact point units 29, 50.Furthermore, electric power for driving motors and actuators inside theinterchangeable lens 2 is also supplied via the contact point units 29,50. For example, lens-specific optical information and informationrelated to a subject distance based on a distance encoder, which arenecessary for the camera control unit 21 to perform focus detection andexposure computation, are output from the lens control unit 51 to thecamera control unit 21 through data communication. Furthermore, focusadjustment information and diaphragm information that have been obtainedas a result of the focus detection and the exposure computationperformed by the camera control unit 21 are output from the cameracontrol unit 21 to the lens control unit 51 through data communication.The lens control unit 51 controls the focusing lens in accordance withthe focus adjustment information, and controls the diaphragm inaccordance with the diaphragm information.

The following describes specific operations from shooting to developmentin the first embodiment. Once the camera control unit 21 is renderedoperable by, for example, turning ON a power switch included in theoperation unit 23 (FIG. 3), the camera control unit 21 first performscommunication with the lens control unit 51 of the interchangeable lens2, and performs initialization processing for, for example, obtaininginformation of various types of lenses necessary for focus detection andmetering. Furthermore, in the operation unit 23, various types of usersettings are accepted, and an arbitrary shooting mode is set. When anoperation of pressing the release switch included in the operation unit23 halfway has been performed, the camera control unit 21 starts theshooting preparation operation, such as AF (autofocus) processing and AE(automatic exposure) processing. Thereafter, when an operation of fullypressing the release switch has been performed, the camera control unit21 performs a shooting operation.

When the shooting operation is performed, light that has passed throughthe interchangeable lens 2 is converted into an electrical signal by theimage sensor 10. Image data generated from this electrical signal isreferred to as a RAW image. Once the RAW image is generated, the signalprocessing circuit 25 performs development processing.

With reference to FIG. 1, a description is now given of the developmentof the RAW image according to the first embodiment. Note that thefunctions of respective units shown in FIG. 1 can be implemented by, forexample, the camera control unit 21, the signal processing circuit 25,or a combination of these.

Each pixel of a RAW image 101 has intensity only in a single colorplane. A white balance unit 102 performs processing for reproducingwhite by correcting a color cast attributed to a light source.Specifically, the white balance unit 102 plots RGB data of each pixel ina predetermined color space, such as an xy color space for example, andresultant G, R, and B of data plotted near a black-body radiation locus,which has a high possibility of representing the color of the lightsource in that color space, are integrated. Then, the white balance unit102 obtains white balance coefficients G/R and GB for an R component anda B component from the integrated value. The white balance unit 102implements white balance processing using the white balance coefficientsgenerated through the foregoing processing.

A color interpolation unit 103 generates a color image in which everypixel has complete RGB color information by performing noise reductionand RAW image interpolation processing. The generated color imageundergoes processing in a matrix conversion unit 104 and a gammaconversion unit 105. As a result, a basic color image (an image to becorrected) is generated (image generation processing). The gammacharacteristic in the case of HDR development in the gamma conversionunit 105 is, for example, the inverse characteristic of the EOTF(Electro-Optical Transfer Function) (FIG. 6) of PQ (PerceptualQuantization). However, as the gamma characteristic, the OOTF(Opto-Optical Transfer Function) characteristic may be combined.

Thereafter, a color/luminance adjustment unit 106 performs processingfor improving the image appearance with respect to the color image.Here, for example, image correction for increasing the brightness in thecase of portrait, enhancing the chroma of green and blue sky in the caseof landscape, and the like is performed. This image correction isexecuted by, for example, applying a lookup table (LUT) forcolor/luminance adjustment to color signal values of RGB and the like.

Furthermore, particularly in the first embodiment, the color/luminanceadjustment unit 106 performs adjustment processing (correctionprocessing) with respect to a luminance component of the color image(regarded here as an I value). The I value is a luminance evaluationvalue calculated from an ICtCp color space in which even ahigh-luminance range that can be presented using HDR can be evaluated.The camera main body 1 holds in advance, as design values 111, aluminance adjustment LUT (reference LUT, second correction information)intended for a shooting mode corresponding to an input/outputcharacteristic (reference input/output characteristic, secondinput/output characteristic) having a peak luminance that serves as areference (a second maximum output luminance value). A difference LUTgeneration unit 112 generates an LUT (difference LUT) that is equivalentto differences from correction amounts in a high-luminance range of thereference LUT in accordance with a peak luminance (first maximum outputluminance value) of an input/output characteristic (selectedinput/output characteristic, first input/output characteristic)corresponding to a shooting mode that is selected at the time ofshooting. The details of processing for generating the difference LUT(third correction information) will be described later. An LUTcomposition unit 113 composites the reference LUT and the differenceLUT, thereby generating a new luminance adjustment LUT (composite LUT)in which the correction amounts in the high-luminance range of thereference LUT have been changed. The details of processing forgenerating the composite LUT (first correction information) will bedescribed later. The color/luminance adjustment unit 106 adjusts(corrects) the luminance values of the color image by applying thecomposite LUT to the color image. Upon completion of processing in thecolor/luminance adjustment unit 106, a compression unit 107 compresses ahigh-resolution image in compliance with the standards of HEVC or thelike. A recording control unit 108 records the compressed image into thestorage unit 26 as a developed image 109.

With reference to FIGS. 10A, 10B, 11A, and 11B, the following describesthe details of the processing performed by the difference LUT generationunit 112 for generating the difference LUT. In step S1000, thedifference LUT generation unit 112 obtains a shooting condition 110(FIG. 1), and determines a shooting mode. The camera main body 1 has ashooting mode that gives priority to high-luminance tone properties (atone priority mode), in addition to a normal shooting mode (a normalmode), and the user selects which one of the shooting modes is to beused during the shooting in accordance with a scene to be shot. Betweenthese two shooting modes, there is a difference in a peak luminance whena shot image has been developed using HDR. Therefore, informationindicating which shooting mode was used during the shooting can be usedto generate the difference LUT. In the following description, it will beassumed that the normal mode corresponds to the aforementioned referenceinput/output characteristic (the second input/output characteristichaving the second maximum output luminance value). It will also beassumed that the tone priority mode has been selected by the user, andthe tone priority mode corresponds to the aforementioned selectedinput/output characteristic (the first input/output characteristichaving the first maximum output luminance value).

In step S1001, the difference LUT generation unit 112 obtains thereference input/output characteristic and the selected input/outputcharacteristic from the design values 111 (FIG. 1). In step S1002, thedifference LUT generation unit 112 obtains the reference LUT (the LUTfor correcting the luminance values of an image to which tone conversionprocessing conforming to the reference input/output characteristic hasbeen applied) from the design values 111 (FIG. 1).

In step S1003, the difference LUT generation unit 112 generates athrough LUT having the same grid as the reference LUT. The through LUTis the LUT in which, as shown in FIG. 11A for example, the same valuesare set under IN (input) and OUT (output) for each grid point (eachinformation portion) of the LUT.

In step S1004, the difference LUT generation unit 112 determines whetherthe number of processed grid points in the through LUT is smaller thanthe total number of grid points. If the number of processed grid pointsis smaller than the total number of grid points, the processing proceedsto step S1005, and if the number of processed grid points is not smallerthan the total number of grid points, the processing of the presentflowchart ends.

In step S1005, the difference LUT generation unit 112 reads out an inputvalue (I value) of one unprocessed grid point in the through LUT. As aresult of repeatedly performing the determination in step S1004 and thereadout in step S1005, all of the grid points in the through LUT areprocessed eventually.

In step S1006, the difference LUT generation unit 112 obtains an inputsignal value by performing a reverse lookup based on the selectedinput/output characteristic with respect to the I value that was readout in step S1005. For example, as indicated by reference sign 1101 ofFIG. 11A, when the I value (IN in the through LUT) that was read out instep S1005 is 18, the input signal value of the selected input/outputcharacteristic is 4.

In step S1007, the difference LUT generation unit 112 obtains, from thereference input/output characteristic, an output value (I value)corresponding to the input signal value that was obtained in step S1006.In the case of the example of FIGS. 11A and 11B, as indicated byreference sign 1102, 16 is obtained as the I value from the referenceinput/output characteristic.

In step S1008, with reference to the reference LUT, the difference LUTgeneration unit 112 obtains a correction amount corresponding to the Ivalue that was obtained in step S1007. In the case of the example ofFIGS. 11A and 11B, as indicated by reference sign 1103, “+2” is obtainedas the correction amount.

In step S1009, the difference LUT generation unit 112 adds thecorrection amount that was obtained in step S1008 to an output value ofthe grid point to be processed in the through LUT (the grid pointcorresponding to the I value that was read out in step S1005). In thecase of the example of FIGS. 11A and 11B, as indicated by reference sign1104, “+2” is added to an output value corresponding to the input signalvalue “18”, thereby associating the input signal value “18” with theoutput value “20” in the through LUT. Note that the value that is addedto the output value in the through LUT here need not necessarily beequal to the correction amount that was obtained in step S1008, as longas the added value is a value based on this correction amount.

In step S1010, the difference LUT generation unit 112 rewrites the inputsignal value of the grid point to be processed (the grid pointcorresponding to the I value that was read out in step S1005) into anoutput value that is obtained by correcting this input signal value inaccordance with the reference LUT. In the case of the example of FIGS.11A and 11B, as indicated by reference sign 1105, the input signal value“18” is rewritten into “19”. As a result, the difference LUT having agrid point at which the input signal value “19” and the output value“20” are associated with each other is generated.

Thereafter, the processing returns to step S1004, and similar processingis repeated with respect to all of the grid points. As a result, theprocessing for generating the difference LUT is completed.

Although the above has described the figures with the assumption ofvariable-grid LUTs by way of example, values can be obtained throughinterpolation from the preceding and succeeding characteristics in thecase of a fixed grid. Furthermore, in a case where a target value doesnot exist at the time of, for example, obtaining a correction amountfrom the reference LUT based on an output value (I value), the value canbe calculated through interpolation processing on an as-needed basis.

Next, with reference to FIGS. 7 to 9, the details of the processingperformed by the LUT composition unit 113 for generating the compositeLUT will be described. As shown in FIG. 7, the LUT composition unit 113generates the composite LUT by applying the difference LUT to outputvalues in the reference LUT. A correction range 72 of the composite LUTthus obtained is larger than a correction range 71 of the reference LUT.Therefore, with use of the composite LUT, correction is applied also toa high-luminance range that is outside the correction range of thereference LUT. FIG. 9 is a diagram showing a difference between thereference LUT and the composite LUT according to the first embodiment.In FIG. 9, a horizontal axis represents an I value, and a vertical axisrepresents a correction amount. A correction amount 91 represents acorrection amount according to the reference LUT, and a correctionamount 92 represents a correction amount according to the composite LUT.As can be understood from the comparison between the correction amount91 and the correction amount 92, with use of the composite LUT,correction is applied also to a high-luminance range that is outside thecorrection range of the reference LUT. Furthermore, regarding aluminance range in which there is no difference in the output luminancevalue, there is no difference in the correction amount, either. FIG. 8Ashows examples of the input/output characteristics corresponding to twotypes of shooting modes with different peak luminances, and FIG. 8Bshows an example of a difference in an output value (I value) betweenthe shooting modes of FIG. 8A. In FIG. 8A, a horizontal axis representsan input signal value, and a vertical axis represents an I value. InFIG. 8B, a horizontal axis represents an input signal value, and avertical axis represents a difference in an I value. Also, it will beassumed that a gamma curve 81 corresponds to the normal mode, and agamma curve 82 corresponds to the tone priority mode. As can beunderstood from FIGS. 8A and 8B, in a high-luminance range in which theinput signal value exceeds a threshold 83, a difference arises in theoutput value (I value). In a region where such a difference arises inthe output value (I value), a difference arises between the referenceLUT and the composite LUT.

As described above, according to the first embodiment, the camera mainbody 1 generates the composite LUT based on differences related tooutput luminance values (differences in output luminance values forrespective input values) between the selected input/outputcharacteristic and the reference input/output characteristic, and on thereference LUT. This enables luminance correction in accordance with themaximum output luminance value of the input/output characteristic oftone conversion processing that was applied to an image to be corrected.

Second Embodiment

The first embodiment has focused on differences in output luminancevalues for respective input values between the selected input/outputcharacteristic and the reference input/output characteristic, asdifferences related to output luminance values between the selectedinput/output characteristic and the reference input/outputcharacteristic. The second embodiment focuses on differences in theinclinations of output luminance values for respective input valuesbetween the selected input/output characteristic and the referenceinput/output characteristic, as differences related to output luminancevalues between the selected input/output characteristic and thereference input/output characteristic. In the present embodiment, abasic configuration of the digital camera is similar to that of thefirst embodiment (see FIGS. 2 and 3). The following mainly describesdifferences from the first embodiment.

With reference to FIG. 12, a description is now given of the developmentof a RAW image according to the second embodiment. In FIG. 12, thedifference LUT generation unit 112 of FIG. 1 is replaced with adifference LUT generation unit 124. Other constituents are similar tothose of the first embodiment. After obtaining a shooting condition 110and design values 111, the difference LUT generation unit 124 generatesa difference LUT in accordance with inclinations calculated from gammadata (an input/output characteristic).

FIG. 13A is a conceptual diagram of gamma curves that respectivelycorrespond to the cases where shooting has been performed in shootingmodes with different peak luminances, and FIG. 13B is a conceptualdiagram of changes in the inclination between the shooting modes at thattime. In FIG. 13A, a horizontal axis represents an input signal value,and a vertical axis represents an I value. In FIG. 13B, a horizontalaxis represents an input signal value, and a vertical axis represents aninclination. A gamma curve 131 corresponds to a shooting mode with apeak luminance that serves as a reference. Here, assume a case whereshooting has been performed in a shooting mode with a peak luminancehigher than that of the gamma curve 131, as indicated by a gamma curve132. Reference sign 133 indicates a change in the inclination of thegamma curve 131, and reference sign 134 indicates a change in theinclination of the gamma curve 132. As shown in here, using an inputsignal value 135 as a threshold, a difference (discrepancy) arises inthe inclination in a range of higher luminances than the threshold.Although the first embodiment has focused on the difference in theoutput value, the difference in the inclination also arises in the sameluminance range. In the cases of the examples of FIGS. 13A and 13B,correction amounts are expanded in a range of luminances higher than theinput signal value 135 in the second embodiment.

With reference to FIGS. 15A, 15B, 16A, and 16B, the following describesthe details of processing performed by the difference LUT generationunit 124 for generating the difference LUT. In FIGS. 15A and 15B, stepsin which processing that is the same as or similar to that of FIGS. 10Aand 10B is performed have the same reference signs as in FIGS. 10A and10B.

In step S1501, the difference LUT generation unit 124 computesinclinations with respect to each of the reference input/outputcharacteristic and the selected input/output characteristic, andassociates them with the respective input/output values. Theinclinations are obtained using the following Expression 1.Inclination=((I value in the second input signal value)−(I value in thefirst input signal value))/((the second input signal value that islarger than the first input signal value)−(the first input signalvalue))  (Expression 1)

In step S1502, using the I value that was read out in step S1005 as anoutput luminance value of the selected input/output characteristic, thedifference LUT generation unit 124 obtains an inclination of theselected input/output characteristic corresponding to the position ofthis output luminance value. For example, as indicated by reference sign1601 of FIG. 16A, when the I value (IN in the through LUT) that was readout in step S1005 is 32, the inclination of the selected input/outputcharacteristic corresponding to the position of the output luminancevalue 32 is 2.

In step S1503, the difference LUT generation unit 124 obtains an outputluminance value that, in the reference input/output characteristic,corresponds to the same value as the inclination that was obtained instep S1502. When a plurality of output luminance values correspond tothe same value as the inclination that was obtained in step S1502, thedifference LUT generation unit 124 selects the smallest value among theplurality of output luminance values. In the case of the example ofFIGS. 16A and 16B, as indicated by reference sign 1602, an output value(I value) 18 corresponding to the inclination 2 in the reference inputcharacteristic is obtained.

Subsequent processing is similar to that of the first embodiment. Thatis, in the case of the example of FIGS. 16A and 16B, in step S1008, “+1”is obtained as a correction amount as indicated by reference sign 1603.In step S1009, as indicated by reference sign 1604, “+1” is added to anoutput value corresponding to an input signal value “32”, therebyassociating the input signal value “32” with an output value “33” in thethrough LUT. In step S1010, as indicated by reference sign 1605, theinput signal value “32” is rewritten into “32” (in the case of thisexample, the numeric value is the same before and after the rewrite). Asa result, the difference LUT having a grid point at which the inputsignal value “32” and the output value “33” are associated with eachother is generated.

Note that similarly to the first embodiment, when the LUTs have fixedgrids, values can be obtained through interpolation from the precedingand succeeding characteristics. Furthermore, in a case where a targetvalue does not exist at the time of, for example, obtaining a correctionamount from the reference LUT based on an output value (I value), thevalue can be calculated through interpolation processing on an as-neededbasis.

Thereafter, the LUT composition unit 113 composites the difference LUTand the reference LUT. As a result, the expansion of the correctionamounts shown in FIG. 14 is executed with respect to the reference LUT.FIG. 14 is a diagram showing a difference between the reference LUT andthe composite LUT according to the second embodiment. In FIG. 14, ahorizontal axis represents an I value, and a vertical axis represents acorrection amount. A correction amount 141 represents a correctionamount according to the reference LUT, and a correction amount 142represents a correction amount according to the composite LUT. As shownin here, also when the difference LUT is generated based on a differencein the inclination of the output luminance value, the correction rangecan be expanded even in a high-luminance range that is outside thecorrection range of the reference LUT, similarly to the firstembodiment. Furthermore, in the second embodiment, the same correctionamount can be used in a luminance range that has the same inclination asthe reference input/output characteristic having a peak luminance thatservers as a reference. Therefore, while a linear region extends furtherin the gamma curve 132 than in the gamma curve 131 in FIGS. 13A and 13B,a constant correction amount can be used in this linear region asindicated by reference sign 143.

OTHER EMBODIMENTS

Although the first embodiment and the second embodiment have beendescribed in relation to luminance correction with the assumption of HDRusing the PQ method, correction amounts can be expanded also by usingthe HLG method with a similar approach. Furthermore, the approachesdescribed in the first embodiment and the second embodiment are notlimited to a composite LUT corresponding to a difference between theshooting modes of HDR, and are also applicable to, for example,processing for generating a composite LUT corresponding to a differencein the peak luminance between SDR and HDR.

Furthermore, in the first embodiment and the second embodiment, at thetime of a shooting mode having an input/output characteristic with ahigh peak luminance, correction amounts are expanded using correctioninformation of a reference shooting mode having an input/outputcharacteristic with a low peak luminance (the reference LUT). However,the above-described configurations are applicable also when themagnitude relationship between these peak luminances is reversed (inthis case, the correction range is reduced in consequence).

Furthermore, although the first embodiment and the second embodiment arebased on the premise of LUTs and input/output characteristics (gammadata) for I values calculated from the ICtCp color space, similarprocessing can be applied also with respect to the RGB color space andthe YUV color space.

Furthermore, although the first embodiment and the second embodiment arebased on the premise of correction processing for luminance components(I values) of a color image, similar processing can be applied also withrespect to processing for color components (chroma and hue). The chromaand hue can be obtained from Expression 2 and Expression 3,respectively, using Ct values and Cp value, which are color componentsof the ICtCp color space. In the case of correction processing for suchcolor components, for example, in the first embodiment, after an I valueis obtained using the same approach until step S1007 of FIG. 10B(corresponding to reference sign 1102 of FIG. 11A), a correction amountfor a Ct value and a Cp value is obtained based on this I value.Although omitted in FIGS. 11A and 11B for the sake of the description ofthe first embodiment, as an LUT for color/luminance adjustment containsan LUT for correction of Ct values and Cp values as well, the same inthe reference LUT is referred to in performing rewriting processing fora through LUT for CtCp.Chroma=√(Ct{circumflex over ( )}2+Cp{circumflex over ( )}2)  (Expression2)Hue=(tan(Cp/Ct){circumflex over ( )}(−1)  (Expression 3)

Furthermore, in the first embodiment and the second embodiment, thedifference LUT is generated by rewriting the input values in the throughLUT after rewriting the output values in the through LUT (see referencesigns 1104, 1105, 1604, 1605 of FIG. 11B and FIG. 16B). However, whenrewriting an output value in the through LUT, a correction amount thatcorresponds to the reference input/output characteristic in thereference LUT may be subtracted from the output value, and then thiscorrection amount may be added to both of an input value and the outputvalue. For example, in the case of the example of FIGS. 11A and 11B, inprocessing indicated by reference sign 1104, the difference LUTgeneration unit 112 obtains a correction amount “+1” that corresponds toan input value 18 in the reference LUT. Then, instead of rewriting anoutput value in the through LUT from 18 into 20, the difference LUTgeneration unit 112 performs a rewrite into “19”, which is obtained bysubtracting the correction amount “+1” from 20. Thereafter, inprocessing indicated by reference sign 1105, the difference LUTgeneration unit 112 adds the correction amount “+1” to each of the inputvalue “18” and the output value “19”, thereby generating the differenceLUT in which the input value “19” is associated with the output value“20”.

Embodiment(s) of the present invention can also be realized by acomputer of a system or apparatus that reads out and executes computerexecutable instructions (e.g., one or more programs) recorded on astorage medium (which may also be referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., central processingunit (CPU), micro processing unit (MPU)) and may include a network ofseparate computers or separate processors to read out and execute thecomputer executable instructions. The computer executable instructionsmay be provided to the computer, for example, from a network or thestorage medium. The storage medium may include, for example, one or moreof a hard disk, a random-access memory (RAM), a read only memory (ROM),a storage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2018-165430, filed Sep. 4, 2018 which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. An image processing apparatus comprising at leastone processor and/or at least one circuit which function as: anobtaining unit configured to obtain a first image to which toneconversion processing conforming to a first input/output characteristichaving a first maximum output luminance value has been applied; ageneration unit configured to generate first correction information forcorrecting a luminance value of the first image based on a differenceregarding an output luminance value between the first input/outputcharacteristic and a second input/output characteristic having a secondmaximum output luminance value, and on second correction information forcorrecting a luminance value of a second image to which tone conversionprocessing conforming to the second input/output characteristic has beenapplied; and a correction unit configured to correct a luminance valueof the first image in conformity to the first correction information,wherein the generation unit generates the first correction informationbased on differences in inclinations of output luminance values, forrespective input values, between the first input/output characteristicand the second input/output characteristic, and on the second correctioninformation.
 2. The image processing apparatus according to claim 1,wherein the generation unit determines a first information portion thatis in the first correction information and is for correcting a firstluminance value based on a second information portion that is in thesecond correction information and is for correcting a second luminancevalue, and the second luminance value is an output luminance value ofthe second input/output characteristic corresponding to an inclinationof the first input/output characteristic corresponding to a position atwhich an output luminance value is the first luminance value.
 3. Animage processing apparatus comprising at least one processor and/or atleast one circuit which function as: an obtaining unit configured toobtain a first image to which tone conversion processing conforming to afirst input/output characteristic having a first maximum outputluminance value has been applied; a generation unit configured togenerate first correction information for correcting a luminance valueof the first image based on a difference regarding an output luminancevalue between the first input/output characteristic and a secondinput/output characteristic having a second maximum output luminancevalue, and on second correction information for correcting a luminancevalue of a second image to which tone conversion processing conformingto the second input/output characteristic has been applied; and acorrection unit configured to correct a luminance value of the firstimage in conformity to the first correction information, wherein thegeneration unit generates the first correction information based ondifferences in output luminance values, for respective input values,between the first input/output characteristic and the secondinput/output characteristic, and on the second correction information,the generation unit determines a first information portion that is inthe first correction information and is for correcting a first luminancevalue based on a second information portion that is in the secondcorrection information and is for correcting a second luminance value,the second luminance value is an output luminance value that, in thesecond input/output characteristic, corresponds to an input value thatcorresponds to the first luminance value in the first input/outputcharacteristic, and the generation unit determines the first informationportion so that a correction amount for the first luminance valueconforming to the first information portion is equal to a correctionamount for the second luminance value conforming to the secondinformation portion.
 4. The image processing apparatus according toclaim 3, wherein the generation unit further generates third correctioninformation that includes a third information portion for correcting athird luminance value into a fourth luminance value, the third luminancevalue being obtained by correcting the first luminance value inconformity to the second correction information, the fourth luminancevalue being obtained by correcting the first luminance value inconformity to the first correction information.
 5. The image processingapparatus according to claim 1, wherein output luminance values of thefirst input/output characteristic and the second input/outputcharacteristic are I values in an ICtCp color space.
 6. An imagecapturing apparatus, comprising: the image processing apparatusaccording to claim 1; and at least one processor and/or at least onecircuit which function as: an image capturing unit; and an imagegeneration unit configured to generate the first image by applying thetone conversion processing conforming to the first input/outputcharacteristic to an image generated by the image capturing unit.
 7. Animage processing method executed by an image processing apparatus,comprising: obtaining a first image to which tone conversion processingconforming to a first input/output characteristic having a first maximumoutput luminance value has been applied; generating first correctioninformation for correcting a luminance value of the first image based ona difference regarding an output luminance value between the firstinput/output characteristic and a second input/output characteristichaving a second maximum output luminance value, and on second correctioninformation for correcting a luminance value of a second image to whichtone conversion processing conforming to the second input/outputcharacteristic has been applied; and correcting a luminance value of thefirst image in conformity to the first correction information, whereinthe generating step generates the first correction information based ondifferences in inclinations of output luminance values, for respectiveinput values, between the first input/output characteristic and thesecond input/output characteristic, and on the second correctioninformation.
 8. A non-transitory computer-readable storage medium whichstores a program for causing a computer to execute an image processingmethod comprising: obtaining a first image to which tone conversionprocessing conforming to a first input/output characteristic having afirst maximum output luminance value has been applied; generating firstcorrection information for correcting a luminance value of the firstimage based on a difference regarding an output luminance value betweenthe first input/output characteristic and a second input/outputcharacteristic having a second maximum output luminance value, and onsecond correction information for correcting a luminance value of asecond image to which tone conversion processing conforming to thesecond input/output characteristic has been applied; and correcting aluminance value of the first image in conformity to the first correctioninformation, wherein the generating step generates the first correctioninformation based on differences in inclinations of output luminancevalues, for respective input values, between the first input/outputcharacteristic and the second input/output characteristic, and on thesecond correction information.
 9. The image processing apparatusaccording to claim 3, wherein output luminance values of the firstinput/output characteristic and the second input/output characteristicare I values in an ICtCp color space.